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ABSTRACT 


The majority of casualties in the ongoing conflicts in Iraq and Afghanistan are 
due to improvised explosive devices (IEDs). To counter this threat, the Marine Corps 
directed that a persistent surveillance capability be identified and fielded as soon as 
possible. As a result, the development and fielding of the Ground Based Operational 
Surveillance System (G-BOSS) occurred rapidly. G-BOSS consists of a tower, multiple 
cameras, and a combat operations center (COC). Today, scores of these systems are in 
use. However, minimal guidance has been provided to operators on effective techniques, 
tactics, and procedures (TTPs). Furthermore, the benefits of adding additional sensors to 
G-BOSS and networking multiple systems are not clear. 

This research investigates these issues through the use of an agent-based 
simulation. Specifically, thousands of computational experiments, utilizing a 
state-of-the-art experimental design, were run on a scenario based on concurrent live 
developmental tests at 29 Palms by the Marine Corps Operational Test and Evaluation 
Activity (MCOTEA). The experiments assessed the ability of the system to correctly 
classify objects (e.g., snipers, IED emplacement, and mortar teams, as well as neutrals 
and friendly forces) over a variety of enemy actions, G-BOSS configurations, and tactical 
choices. The results indicate that the most critical factor in determining the level of 
situational awareness provided by G-BOSS is, by far, placement of the towers. 
Moreover, little benefit is seen in coordinating the towers and COCs unless motion 
detection radars are used. With use of the motion detection radar, the synchronization of 


multiple systems dramatically enhances the overall performance of G-BOSS. 
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THESIS DISCLAIMER 


The reader is cautioned that the computer programs presented in this research may 
not have been exercised for all cases of interest. While every effort has been made, 
within the time available, to ensure that the programs are free of computational and 
logical errors, they cannot be considered validated. Any application of these programs 


without additional verification 1s at the risk of the user. 
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EXECUTIVE SUMMARY 


The majority of casualties in the ongoing conflicts in Iraq and Afghanistan are 
due to improvised explosive devices (IEDs). To counter this threat, coalition forces have 
directed that a persistent surveillance capability be identified and fielded as soon as 
possible. As a result, the Ground Based Operational Surveillance System (G-BOSS) was 
developed and fielded. G-BOSS consists of a tower, multiple cameras, and combat 
operations centers (COCs). Today, hundreds of these systems are in use. However, 
minimal guidance has been provided to operators on effective techniques, tactics, and 
procedures (T'TPs). Furthermore, the services are unsure of the benefits of adding 
additional sensors to G-BOSS and networking multiple systems. 

This research provides Coalition forces with analytical support for initial and 
further development of TTPs, modernization efforts, and operational employment for 
G-BOSS. The analysis is guided by three questions from the Marine Corps Operational 
Test and Evaluation Activity (MCOTEA), who is tasked with testing G-BOSS in order to 
enhance the operational effectiveness of the system. The questions are: 


@ What critical factors determine the level of situational awareness (SA) 
provided by G-BOSS” 


6 Does coordination via the COC improve the effectiveness of G-BOSS? 


® Does use of motion detection radar improve the effectiveness of G-BOSS? 


This thesis uses agent-based simulation, state-of-the-art design of experiments, and 
graphical and statistical analysis methods to investigate these questions. 

The goal of the simulation is to measure the level of SA provided by G-BOSS. 
The measures of effectiveness (MOEs) associated with this goal are the proportion of 
correct classifications of friendly, hostile and neutral role players per test trial. An 
additional MOE is whether or not the IED emplacers were detected. Factors of interest 
include, but are not limited to: the slew rate of G-BOSS, the distance between the pairs 
of Rapid Aerostat Initial Deployment (RAID) towers, the configuration of the COCs 
(coordinated or stand-alone), and the presence of Man-Portable Surveillance and Target 


Acquisition Radar (MSTAR) with the G-BOSS sensors. 


X1X 


The simulated scenario in this study models a live developmental test conducted 
by the Marine Corps Operational Test and Evaluation Activity (MCOTEA) on the 
capabilities and limitations of G-BOSS conducted in April 2008 in 29 Palms, California. 
The scenario used in this thesis is test trial #15, an IED scenario conducted at night. Trial 
#15 includes an JED-emplacement team, composed of three hostile role players 
simulating IED emplacement at a predetermined time and location. The remaining 
neutral, friendly and hostile forces participating in the trial (also consisting of three 
members each) maneuver through the test area while G-BOSS operators use the system 
to determine hostile acts or intent. Figure ES-1 shows the terrain and agent depictions 


used in the simulation model. 
PARA eal S22 
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Figure ES-1. MANA model screen shot. 


Map Aware Non-Uniform Automata (MANA) is the modeling environment used 
in this thesis. MANA represents the key temporal and spatial elements of this tactical 
scenario and facilitates quickly constructing “medium-resolution” simulations that can be 
broadly explored using sophisticated design of experiments and computing clusters. 
Nearly Orthogonal Latin Hypercubes (NOLHs) and data farming enabled an analysis of a 


large set of possibilities. The conclusions in the research are based on over twenty 


XX 


thousand simulated tests with varying threats (1.e., snipers and IED emplacement teams) 


neutrals and friendly forces over a variety of enemy actions, G-BOSS configurations, and 


tactical choices. 


The insights gained in this analysis are: 


The positioning of the towers is the most critical factor associated with 
enhancing the operational effectiveness of G-BOSS. A distance of 4 km 
between G-BOSS towers results in a proportion of correct identifications 
of 0.91 while a distance of 2 km results in a proportion of 0.53. When 
G-BOSS is employed in open terrain, more dispersion results in 
better performance. 


The stealth of the role-players has a significant effect on the proportion of 
correct identifications. This result is intuitive and helps validate the 
model, since snipers and IED emplacers tend to use stealth to mask their 
hostile acts or intent. Increased training of G-BOSS operators’ level of 
vigilance is a countermeasure to mitigate enemy stealth. 


Without the presence of MSTAR, the coordinated G-BOSS configuration 
produces a slightly larger proportion of correct identifications than the 
stand-alone G-BOSS configuration. The coordinated G-BOSS 
configuration results in an overall proportion of correct identification of 
0.51, while the stand-alone G-BOSS configuration results in an overall 
proportion of correct identification of 0.49. The results of the comparison 
between the G-BOSS configurations are not practically significant. This 
is based on the marginal increase of approximately two percentage points 
between the coordinated G-BOSS configuration and the stand-alone G- 
BOSS configuration. Further, this finding answers the question of 
underutilization: Stand-alone employment of G-BOSS 1s nearly as 
effective as a coordinated employment of G-BOSS without MSTAR. 


With MSTAR, the coordinated G-BOSS configuration produces a 
significantly larger proportion of correct identifications than what the 
stand-alone configuration produces. The coordinated G-BOSS 
configuration results in an overall proportion of correct identifications of 
0.71 and the stand-alone configuration results in an overall proportion of 
correct identifications of 0.47. This result is practically and statistically 
significant. MSTAR facilitates the detection of more agents. MSTAR 
coupled with data fusion at the COC affords commanders an 
enhanced capability. 


The emplacement of the IED was detected in 76% of all of the simulation 
excursions conducted (which varied 13 factors associated with G-BOSS or 
the role-players). This is promising since G-BOSS’s mission is to counter 
the threat of IEDs. 
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I. INTRODUCTION 


A. MOTIVATION 


The improvised explosive device (IED) has proven effective against Marines in 
Operation Iraqi Freedom (OIF). As a result, the largest number of Marines killed in 
action (KIA) and/or wounded in action (WIA) during OIF is attributed to the IED. In 
addition to IEDs, insurgents have expanded their range of tactics, techniques, and 
procedures (TTPs) against Marine forces in Iraq with sniper attacks and hit and run 
indirect fire (IDF) teams. The potential to cause high-density, melodramatic attacks to 
test the resolve of our nation in fighting the Global War on Terror is the central value of 
the IED to insurgents. 

In an attempt to mitigate further casualties by these popular tactics, the 
Marine Corps Combat Development Command (MCCDC), Capabilities Development 
Directorate (CDD), and Capabilities Development Integration Board (CDIB), under the 
guidance of the Marine Corps’ Intelligence, Surveillance, and Reconnaissance (ISR) 
Strategy, directed that a persistent surveillance capability be identified and fielded 
immediately. The ISR strategy 1s a component of the Marine Corps ISR-Enterprise 
(MCISR-E), whose focus is to integrate air, ground, and space sensors into a network 
capable of detecting, locating, identifying, and targeting threats. 

The Ground-Based Operational Surveillance System (G-BOSS)—the focus of this 
thesis—is one system within a larger Family of Systems (FoS) predicated on the concept 


of the Marine Corps’ ISR strategy. 
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Figure 1. G-BOSS operational construct. 


G-BOSS is a force protection, camera-oriented, day/night, expeditionary tool that 
provides the ability to detect, track, display, record, assess, deny, and store video to 
counter the threat of IEDs and disrupt insurgency activities (Marine Corps Combat 
Development Command, 2007) as shown in Figure 1, the G-BOSS operational construct. 
G-BOSS referred to throughout this analysis is composed of a Rapid Aerostat Initial 
Deployment (RAID) 107-foot mobile tower, two cameras: a Star SAFIRE IIIFP and a 
T-3000, a Man-Portable Surveillance and Target Acquisition Radar (MSTAR) sensor, 
and a Ground Control Station (GCS). 


= 





Figure 2. G-BOSS components from I-r: RAID tower, T-3000, Star SAFIRE IIIFP, 
RGS, MSTAR. (from MCOTEA’s G-BOSS Detailed Assessment Plan, April 2008 
[Best viewed in color]) 
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B. THE PROBLEM 


The objective for G-BOSS is to detect, identify, and track insurgent activities; 
specifically those associated with the emplacement of IEDs. Since its initial employment 
in late spring 2007, G-BOSS has contributed to the reduction of the loss of life among 
Marine forces due to IEDs. One potential end-state is a fully networked G-BOSS capable 
of integration with unmanned aircraft systems (UAS), command and control (C2) assets, 
and fire support systems that will increase situational awareness (SA) and aid 
commanders in conducting current and future offensive operations to disrupt 
insurgent activity. 

G-BOSS is currently deployed in a stand-alone configuration with minimal 
centralized coordination and consolidation of sensor data. Employment of G-BOSS ina 
stand-alone configuration is not due to fault, nor negligence. The immediate need for G- 
BOSS, coupled with an aggressive implementation of the system in theater, did not afford 
time to develop TTPs for G-BOSS. The Marine Corps has a four-phase employment plan 
for G-BOSS. The Marine Corps is in phase two of its four-phase employment of G- 
BOSS. The phases are: 


— Phase l 

° Manual operation 

° View video at base of each tower 

° Radio communication to a combat operations center (COC) 
—~ Phase 2 

° Manual operation 


° Video feed to COC 


° COC directs camera slewing 
- Phase 3 
° Cameras controlled from COC 
8 Automatic camera slewing 
- Phase 4 
e Integrated network throughout province (multiple COCs) 


The Marine Corps Systems Command (MCSC), Counter-Improvised Explosive 
Devices Technology Directorate (C-IED) wants to know if the G-BOSS employment 
approach in Phase 2 provides more SA than the G-BOSS employment approach in 
Phase 1. This thesis addresses the question: How can Marines deployed now best 
employ G-BOSS? The Marine Corps Operational Test and Evaluation Activity 
(MCOTEA) conducted a developmental test of G-BOSS to evaluate the capabilities and 
limitations of the system, in order to enhance its operational effectiveness for deployed 
commanders. Results from this test are intended to improve the phased deployment 


of G-BOSS. 


C. PURPOSE 


The purpose of this thesis is to develop a simulation using an agent-based model 
(ABM) that represents the live developmental test conducted by MCOTEA. The goal is 
to provide coalition forces with analytical support to improve G-BOSS TTPs and enhance 
the operational effectiveness of the system. The simulated operational experience 
provided by this analysis helps facilitate future tests of G-BOSS capabilities and can 
potentially serve as an operational planning tool for commanders. 

Further, this study serves as a proof of concept of the capability that the 
Simulation Experiments and Efficient Design (SEED) Center for Data Farming at the 
Naval Postgraduate School (NPS) can provide to support the Joint Improvised Explosive 
Device Defeat Organization (JIEDDO) in the war against IEDs. The SEED Center for 
Data Farming mission statement is to “advance the collaborative development and use of 
simulation experiments and efficient designs to provide decision makers with timely 
insights on complex systems and operations.” JIEDDO has the mission to “lead, 
advocate, and coordinate all Department of Defense actions in support of Combatant 
Commanders’ and their respective Joint Task Forces’ efforts to defeat improvised 
explosive devices as weapons of strategic influence.” The SEED _ Center 
(http://harvest.nps.edu) and JIEDDO (https://www.jieddo.dod.mil/) websites provide 


more detail. 


D. RESEARCH QUESTIONS 


This thesis simulates MCOTEA’s live test and conducts a comparative, 
quantitative analysis of G-BOSS employment TTPs to enhance the operational 
effectiveness of G-BOSS. While this analysis is by no means exhaustive, the following 


questions are addressed: 


@ What critical factors determine the level of SA provided by G-BOSS” 
e Does coordination via the COC improve the effectiveness of G-BOSS? 


e Does use of motion detection radar improve the effectiveness of G-BOSS? 


E. METHODOLOGY 


This thesis uses agent-based simulation, state-of-the-art design of experiments, 
and advanced data analysis methods to analyze the critical factors associated with the 
level of SA provided by G-BOSS. The process is to simulate a scenario based on a live 
developmental test conducted by MCOTEA with G-BOSS. The simulated model is then 
replicated and analyzed. The analysis process uses a technique called data farming, 
which involves using high-performance computing to run the simulation thousands of 
times, while simultaneously varying many input parameters. As a result, data farming 
provides insights into complex problems through the exploration of a multitude of 
possible outcomes. 

This thesis uses an agent-based distillation, which is a type of computer 
simulation that attempts to capture the critical factors of interest in combat without 
explicitly modeling all of the physical details. The tool used is Map Aware Non-Uniform 
Automata (MANA), developed by the New Zealand Defense Technology Agency (DTA). 
More information about MANA can be found at https://teams.nzdf.mil.nz/sites/mana/ 


default.aspx. 


F. THESIS ORGANIZATION 


Chapter II begins with an unclassified description of the equipment and the 


scenario involved with MCOTEA’s test. The chapter concludes with an overview of 


MANA and a detailed description of the simulation model created for this thesis. 
Chapter HI offers a discussion on the design of experiments used for this analysis and 
includes a description of the factors used in this analysis, and an explanation of 
Nearly Orthogonal Latin Hypercubes (NOLHs). Chapter IV provides a description of the 
analytical methods used to interpret the results of the simulated tests and concludes with 
an explanation of the analytical results. Chapter V completes the thesis with a discussion 
of the insights gained from the analysis and recommends topics _ for 


follow-on research. 


HT. MODEL DEVELOPMENT 


Persistent surveillance is the primary capability afforded to the commander by 
G-BOSS. Under the guidance of the Marine Corps’ ISR strategy, G-BOSS will be 
employed at all levels of the Marine Air Ground Task Force (MAGTF) to perform 
surveillance of assigned areas of responsibility (AOR). G-BOSS is now employed in a 
stand-alone configuration. MCSC wants to know if current employment techniques 


underutilize G-BOSS. MCOTEA’s test of G-BOSS addresses that question. 


A. G-BOSS TEST OVERVIEW 


MCOTEA’s developmental test of G-BOSS evaluates its capabilities and 
limitations. Their assessment included three distinct test events: SA; Detection, 
Classification, Recognition, and Identification (DCRI); and Track Error. The scenario 
and simulation model developed for use in this thesis depicts one night time test trial of 
the SA test event. The SA test event explores whether a coordinated network approach of 
G-BOSS significantly improves SA over the current uncoordinated, stand-alone 
tower approach. 

MCOTEA measures SA and compares the level of SA between a coordinated and 
stand-alone tower approach in two steps. First, test trials that use role players in teams of 
friendly, hostile and neutral personnel to maneuver at varying ranges within G-BOSS’s 
field-of-regard. The G-BOSS operators use the system and determine hostile acts or 
intent. The second part is to calculate an SA score based on an SA Value Model 
developed by MCOTEA. Table 1 lists the data requirements necessary to calculate the 
SA score. To learn more about the MCOTEA Situational Awareness Value Model, 
contact MCOTEA at www.quantico.usmc.mil/activities/?Section=MCOTEA for a copy 
of the report detailing the results of the developmental test for G-BOSS. 


MSTAR Percent of Detections 


Enemy Action Percent of False Detections 


Lux Parget Location Error 
Probability of Identification|Cue __[Time to Respond to Cue 


Table 1. Data requirements for MCOTEA’s SA value model. 





1. Test Location 


The test was conducted at Acorn Range, Marine Corps Air Ground Combat 
Center, 29 Palms, California, from 15-28 April 2008. In Figure 3, the red colored area 
depicts the zone in which teams of role players maneuvered against G-BOSS during the 


SA test event. 








Figure 3. G-BOSS test site. (from MCOTEA’s G-BOSS Detailed Assessment Plan, 
April 2008 [Best viewed in color]) 


Zs Test Equipment 


Due to the classified nature of the exact specifications and capabilities of 
G-BOSS, only a general description is provided. Figure 2, in Chapter I, illustrates 
G-BOSS undergoing test and evaluation. The Star SAFIRE HIFP and T-3000 cameras 
are mounted at the top of the RAID tower. A tactical radio for the transmission of 
camera and radar data is mounted near the cameras. MSTAR is mounted on the tower, 
approximately 28 feet above the tower’s base. MCOTEA assessed (4) RAID towers, 
(8) cameras, (4) MSTARs, (4) GCSs, (2) RGSs, and (2) COCs. 


3. Test Architecture 


Figure 4 shows the test equipment configuration: four RAID 107-foot towers, 
each configured with one Star SAFIRE IIIFP camera and one T-3000 camera. The 
G-BOSS sensors were deployed in two distinct configurations: a coordinated network 
configuration and an uncoordinated, stand-alone configuration. Each tower pair (1 and 2, 
3 and 4) spans a 2-km distance. For administrative and logistical convenience, tower 
pairs were located near each other. COC 1 and COC 2 are located approximately 50 km 


away from their respective towers. 





Figure 4. MCOTEA test architecture. 


Towers | and 2, along with COC 1, were operated in a coordinated network 
configuration. The coordinated network approach allows COC I to optimize the search 


patterns of the four G-BOSS sensors on Towers | and 2. Under this configuration, 
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COC | is able to assist Towers | and 2 assess hostile acts or intent during the test trial. 
COC 1 is able to assist Towers 1 and 2, since it can observe what all four G-BOSS 
sensors on Towers | and 2 can observe. 

Towers 3 and 4, along with COC 2, were operated in an uncoordinated, 
stand-alone configuration. This configuration allows degraded assistance from COC 2 to 
Towers 3 and 4 during the test trial to assess hostile acts or intent. COC 2 could not 
observe what Towers 3 and 4 could see, but received updates via voice over internet 


protocol (VOIP) phone transmission. 


4. Situational Awareness (SA) Trial 


An SA trial consisted of a predetermined time period wherein a mix of friendly, 
neutral and hostile role players operated on foot in the field of view of the G-BOSS 
towers. During this period, some of the hostile role players performed their mission. A 
trial period starts at either the beginning of a test day or at the end of the previous data 
collection stop. A trial period ends at a predetermined time for a data collection stop. 
The two distinct tower and COC configurations operate simultaneously during the trial, 
so, at a minimum, each data collection stop yields two trials. 

The SA trial associated with this thesis was test trial #15, which was an IED 
scenario conducted on 24 April 2008 at night. The scenario included an 
IED-emplacement team, composed of three hostile role players simulating IED 
emplacement at a predetermined time and location. The remaining neutral, friendly and 
hostile forces participating in the trial (also consisting of three members each) moved 
through the test area as directed by the test-trial script. The trial concluded upon 
completion of the hostile act and after MCOTEA called for a data collection stop. 
G-BOSS operators were provided a table, such as Table 2, to facilitate distinguishing 


between role players. 
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Neutral SN Middle Easter Clothing 
(N) 


Friendly Helmet/Flak w/Mortar Tube 
(F) F-Rifle/Scope Helmet/Flak w/Rifle/Scope 


Helmet/Flak w/Rifle 
Helmet/Flak 


Hostile H-Shovel/IED Middle Eastern Clothing w/Shovel/IED 
(H) H-Rifle/Scope Middle Eastern Clothing w/Rifle/Scope 
Middle Eastern Clothing w/Mortar Tube 


Table 2. SA test trial role-player categories. 





B. THE MAP AWARE NON-UNIFORM AUTOMATA (MANA) COMBAT 
SIMULATION TOOL 


MANA is the modeling environment used in this thesis. MANA is a time-step, 
ABM environment developed by the New Zealand DTA in early 2000 for the 
New Zealand Defense Force (NZDF) after experiencing frustration with combat models 
based solely on physics. MANA intends to capture “enough physics as is necessary” 
(McIntosh, Galligan, Anderson, & Lauren, 2007), to capture aspects of human behavior 
and to represent a wide range of interactions among agents and their environment. Ina 
MANA model, the agents are: 

® Map Aware—Agents have SA of the other agents and terrain that is 

updated by sensors and communications. 


8 Non-uniform—Individual agents may have different behavior parameters, 
capabilities, sensors, weapons, and communications. 


° Automata—Agents react independently on the battlefield according to 
their own individual characteristics and awareness. 


MANA is a straightforward application that is intuitive and easy to use with a 
well-developed Graphical User Interface (GUI). More details of this model are readily 
available in the MANA User’s Manual and at https://teams.nzdf.mil.nz/sites/mana/ 


default.aspx. 
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1. | Why MANA? 


Many aspects of warfare can be explained by physics, such as the trajectory of a 
projectile to a point of impact or the effective casualty radius of an artillery round. Less 
tangible elements, such as SA, must also be explained or measured. The nonlinear 
relationships associated with SA, coupled with the numerous variables involved, make 
sole dependence on physics-based models ineffective. 

MANA represents the key temporal and spatial elements of the tactical scenario. 
With a clear idea of this scenario and the measure in question, MANA can provide the 
capability to “explore the greatest range of possible outcomes with the least set-up time” 
(McIntosh, Galligan, Anderson, & Lauren, 2007). 

MANA is used because SA 1s the primary measure to be analyzed. In the most 
basic of terms, “SA is knowing what is going on around you” (Endsley & Garland, 2000). 
Many factors influence the various levels of SA. MANA provides a dynamic and 
stochastic environment in which many of these factors can be varied and _ their 
interactions explored. MANA also facilitates rapid scenario generation based on 


MCOTEA’s developmental test. 


C. CHARACTERISTICS OF THE SIMULATION MODEL 


The role players and equipment associated with the SA test trial are referred to as 
agents. An agent is a “character” within the MANA modeling environment with assigned 


attributes and personality characteristics similar to their actual capabilities. 


Detection is defined as one agent being aware of another agent’s presence. 
Classification is a step further. Classification means an agent is able to distinguish if a 
detected agent is a neutral, hostile or friendly agent. In the MCOTEA test, detection 
takes on the same definition as in the MANA scenario. Identification in the MCOTEA 


test is the equivalent to classification in the MANA scenario. 


1. Goal 


The goal of the simulation is to measure the level of SA provided by G-BOSS for 


test trial #15. The measures of effectiveness (MOEs) associated with this goal are the 
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proportion of correct classifications of friendly, hostile and neutral role players per test 
trial. An additional MOE is whether or not the IED emplacers were detected. Factors of 
interest include, but are not limited to: the slew rate of G-BOSS, the distance between 
the pairs of RAID towers, the configuration of the COCs (coordinated or stand-alone), 
and the presence of MSTAR with the G-BOSS sensors. 


2. Scale and Terrain 


The implementation of the scenario in MANA requires mapping from real time 
and real space to simulated time and simulated space. In this simulation model, a time 
step is equal to 5 seconds of real time. For a 1-hour scenario this equates to 720 model 
time steps. The default battlefield size for a scenario in MANA is a 200 x 200 grid. The 
scale of the scenario used in the simulation model is 20 meters per grid square. Thus, a 
650 pixel by 500 pixel map grid represents the 13 kilometer by 10 kilometer test area 
used in the MCOTEA test. Due to the few agents needed for the simulation, a 1-hour 
simulated test trial completes in approximately 30 seconds of real time per execution on a 
standard laptop processor. The time and distance scales were chosen to achieve “good 
enough” resolution to simulate and record the result of agent activities. 

The terrain in the model consists of two components: an elevation map and a 
terrain map as shown in Figure 5. In the elevation map, black represents the lowest level 
of elevation and white represents the highest elevation. Elevation in MANA is used to 
block line-of-sight, as appropriate. To obtain the elevation map, the Falcon View 
software package was used to open the appropriate DTED-2 file that was obtained from 
the National Geospatial-Intelligence Agency (NGA). The image file acquired with 
Falcon View was then translated into a bitmap that could be read in by MANA. Finally, 
within MANA, the user manually enters the difference between high and low points in 
the image (which corresponds to the black to white spectrum MANA uses to display the 


elevation file). 
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MANA Elevation File 
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Terrain from MCOTEA’s Test 


MANA Terrain File 





Figure 5. The elevation and terrain files used by MANA were derived from the 
terrain in the MCOTEA scenario. [Best viewed in color] 


The terrain file depicts battlefield terrain in the model. Distinct colors are used to 
represent terrain features within the model. Figure 6 shows how colored pixels on the 
terrain map (as defined by their Red-Green-Blue settings) affect the movement speed of 
the role-players. Figure 6 also shows the level of cover and concealment provided by the 
terrain. For example, a terrain type created for this scenario called defilade, which is 
depicted by the brown spots in the MANA terrain file, allows only 70% of the maximum 
speed for a role-player, 10% protection from fire, and 60% concealment from view. 
These percentages are used as multipliers in MANA’s calculations, and their use is 


further described in the MANA User’s Manual. 
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Figure 6. The characteristics of the colors in the terrain file for the MANA model 
used in this research. (from MANA 4.01.1) 


3. Agent Descriptions 


a. The Refueler Agent Class 


A large portion of the test trial depends upon the agent’s stealth as it 
maneuvers in the test area. To facilitate stealth, an agent named “Refueler” was created 
to serve the purpose of “refueling” the role-player agents in the scenario, according to its 
range and probability settings. Receiving fuel from the refueler triggers the agent to 
temporarily go into a stealth state where it stops moving for 12 time steps (1 minute). 
The refueler agent is invisible to other agents and provides fuel every 36 time steps 


(3 minutes) on average. 


4. Red Agents (Hostile Role Players) 


SA test trial #15 uses two types of hostile role-players: an JED emplacer and a 
sniper. These agents are referred to as H-ShovellED and H-RifleScope, respectively. 
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a. H-Shovel[ED 


This agent is depicted by a standing red soldier icon on the screenshot 
shown in Figure 7. This agent moves slowly and stealthily, while seeking defilade 
positions, until it arrives at the IED emplacement location. During its hostile act, its 
appearance changes into a larger red soldier icon. The simulated hostile act takes place at 
approximately time step 360 (30 minutes into the scenario, in accordance with the live 
test event). The act continues for approximately 120 time steps (10 minutes). During the 
simulated hostile act, the agent loses the stealth property because it is digging. When it 1s 
refueled by the Refueler, this agent stops moving and employs stealth for 12 time steps 


(1 minute). 
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Figure 7. MANA Model Screen Shot. [Best viewed in color] 


b. H-RifleScope 


This agent is depicted by a red soldier icon in the prone position on the 
screenshot shown in Figure 7. This agent moves slowly and stealthily seeking cover and 
concealment from defilade to its final hide position. When it is refueled by the Refueler, 


this agent stops moving and employs stealth for 12 time steps (1 minute). 
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>; Blue Agents (Friendly Role Players) 


SA test trial #15 uses two types of friendly role-players: a Marine with a rifle and 
a scope and a Marine with just a rifle. These agents are referred to as 


F-RifleScope and F(Friendlies), respectively. 


a. F-RifleScope 


This agent is depicted by a standing blue soldier icon on the screenshot 
shown in Figure 7. This agent moves slowly and stealthily seeking cover and 
concealment from defilade to its final hide position. When refueled by the Refueler, this 


agent stops moving and employs stealth for 12 time steps (1 minute). 


b. F (Friendlies) 


This agent is depicted by a standing blue soldier icon on the screenshot 
shown in Figure 7. This agent moves slowly and stealthily seeking cover and 
concealment from defilade to its final hide position. When refueled by the Refueler, this 


agent stops moving and employs stealth for 12 time steps (1 minute). 

6. Neutral Agents (Neutral Role Players) 

SA test trial #15 uses one type of friendly role-player: a local member of the area. 
This agent 1s known as N(Neutrals). 


a. N(Neutrals) 


This agent is depicted by a standing green soldier icon on the screenshot 
shown in Figure 7. This agent maneuvers in the test area without seeking cover and 
concealment. This agent does not turn yellow since there is no need for it to be stealthy. 


This agent simply wanders about the test area. 


Ts Towers 1 and 2 


SA test trial #15 uses two towers with the traits listed below. These agents are 


referred to as Tower | and Tower 2. 
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a. Tower 1 and Tower 2 


These agents are depicted by purple tower icons on the screenshot shown 
in Figure 7. These agents do not move. They detect and classify other agents in the test 
area with their three sensors, T-3000, StarSAFIRE IIIFP, and MSTAR. The agents 
continually scan the test area to classify other agents, and once the tower classifies 
another agent in the test area, the tower “shoots” a round from its perfectly lethal, 
perfectly accurate weapon. In the model, shooting/killing is a surrogate for classifying. 
This is done to simplify the MOE data collection. This approach is reasonable since 
combat adjudication is not employed. At least three discrete time observations 
(15 seconds of observation or shots) are required to classify an agent. 

Towers | and 2 along with COC | represent the coordinated network 
configuration described in the test architecture. To model that configuration, all 
classification information is passed instantaneously and accurately to COC 1. This 


approach simulates four monitors, one for each view of each tower’s cameras, in COC 1. 


8. Towers 3 and 4 


SA test trial #15 uses two towers with the same traits. These agents are known as 


Tower 3 and Tower 4. 


a. Tower 3 and Tower 4 


These agents are depicted by purple tower icons on the screenshot shown 
in Figure 7. These agents do not move. They detect and classify other agents in the test 
area with their three sensors, T-3000, StarSAFIRE IIIFP, and MSTAR. The agents 
continually scan the test area to classify other agents, and once the tower classifies 
another agent in the test area, the tower “shoots” a round from its perfectly lethal, 
perfectly accurate weapon. In the model, shooting/killing is a surrogate for classifying. 
This is done to simplify the MOE data collection. This approach is reasonable since 
combat adjudication is not employed. At least three discrete time observations (or shots) 
are required to classify an agent. 

Towers 3 and 4 along with COC 2 represent the stand-alone network 


configuration described in the test architecture. To model that configuration, the Towers 
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attempt to pass all classification information to COC 2; however, with delays and 
inaccuracies. Only one message at a time is passed. This technique simulates the 


G-BOSS limitation of only monitoring one threat at a time. 


9, COC 1 and COC 2 


SA test trial #15 uses two COCs with different traits. These agents are called 
COC 1 and COC 2. 


a. COC 1 


This agent is depicted by a purple crosshair icon on the screenshot shown 
in Figure 7. This agent does not move and has no organic sensor associated with it. 
COC 1 receives classification information from Towers 1 and 2, in a manner that 
simulates two monitors associated with each tower’s cameras. COC 1 performs data 
fusion on the information received from Towers | and 2 to assist the towers with 
classification of agents. For more detail on MANA data fusion, the MANA User’s 
Manual may be consulted. The data fusion capability is used as a surrogate to model the 
effect of the COC operators assisting with the classification task. COC classification is 


accomplished in the same manner as classification by a tower. 


b. COC 2 


This agent is depicted by a yellow crosshair icon on the screenshot shown 
in Figure 7. COC 2 operates in a similar manner to COC 1. COC 2 receives less 
information from Towers 3 and 4. Only completed classifications are passed from 


Towers 3 and 4 to COC 2. COC 2 does not employ data fusion. 
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Hl. EXPERIMENTAL DESIGN 


A goal of this thesis is to provide insights that enhance the operational 
effectiveness of G-BOSS. Experimental design is used to identify the critical factors 
associated with the level of SA provided by G-BOSS. The experimental design includes 
selection of factors and measures of effectiveness (MOEs). State-of-the-art techniques 
are applied to efficiently explore the parameter space through data farming. NOLHs 
facilitate this approach. 


A. MEASURE OF EFFECTIVENESS 


A MOE is “a measure of operational success that must be closely related to the 
objective of the mission or operation being evaluated. A meaningful MOE must be 
quantifiable and a measure to what degree the real objective is achieved” (The Defense 
Acquisition University [DAU], 2003). SA is the intangible measure in the center of this 
analysis. A simple definition of SA is “knowing what is going on around you” (Endsley 
& Garland, 2000). SA also includes observing the battlespace, processing input from 
those observations and developing an understanding of the environment, friendly actions 
and threat activity. Two MOEs are used to quantify SA in this analysis: the percent of 
correct classifications made by G-BOSS operators and whether or not an IED 


emplacement is observed. 


B. EXPERIMENT FACTORS 


The simulation factors chosen for this thesis are based on the developmental test 
conducted by MCOTEA on G-BOSS. The experiment factors are grouped into three 
categories: G-BOSS configuration, G-BOSS performance, and battlespace environment. 
G-BOSS configuration and G-BOSS performance factors can be controlled in the real 
world by the decision maker. For example, the distance between each G-BOSS RAID 
tower 1s dictated by the COC. Battlespace environment factors cannot be controlled in 
the real world by the decision maker. Examples include the number of hostile, friendly, 
or neutral personnel in a G-BOSS field-of-regard. Table 3 summarizes the simulation 


parameters and their ranges used in the experiment. 
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Cam 1 P(Class) 01....99 Probability of classification associated with the T-3000 
sensor 
Cam 1 Class Range +/- 20% Probability of classification range associated with the T- 
3000 sensor 
Probability of classification associated 
iia chan head with the Star SAFIRE IIIFP sensor 
Probability of classification range associated with the Star 
Cam 2 Class Range +/- 20% SAFIRE IIIFP sensor 
Tower Slew Rate 1-3 deg/ts Surrogate for time it takes to cover the area in high- 
resolution mode 
Latency of Comm to COC 1 min... Delay associated with passing contact info from tower to 
15 min COC 
Reliability of Comm to COC |_ .01....99 Probability that intended message from tower is received 
by COC 
nl ae ees ee Number of role-players per team 


Role-player Stealth .01....99 Level of stealth associated with each role-player 


5 km/hr to 
Role-player Speed Doin Maneuver speed for each role-player 


Table 3. Variable factors in the experimental design. G-BOSS configuration factors are in 
yellow, G-BOSS performance factors are in white, and Battle space environment 
factors are in gray. [Best viewed in color] 





1. G-BOSS Configuration Factors 
The following factors are used to analyze the level of SA _ provided 
by G-BOSS: 
a. MSTAR or No MSTAR 


This is a categorical variable defined by whether the motion detection 


radar is present or not. 


b. Number of COC Monitors 
This is the number of monitors in the COC. 
C. Distance Between Pairs of Towers 


This is the distance between RAID towers. 
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2. G-BOSS Performance Factors 


The following factors are used to analyze the level of SA provided by 


G-BOSS on a basis of performance: 


a. Camera I P(Classification) 
This is the probability of classification associated with the T-3000 sensor. 
b. Camera I Classification Range 


This is the probability of classification range associated with the 


T-3000 sensor. 


C: Camera 2 P(Classification) 


This is the probability of classification associated with the Star SAFIRE 
IIIFP sensor. 


d. Camera 2 Classification Range 


This is the probability of classification range associated with the Star 


SAFIRE IIIFP sensor. 


é. Tower Slew Rate 


This is a surrogate for time it takes to cover the area in 


high-resolution mode. 


fs Latency of Communication to COC 


This is the delay associated with passing contact information from the 


tower to the COC. 


g. Reliability of Communication to COC 


This is the probability that messages from the tower are received by 


the COC. 
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3. Battle Space Factors 


The following factors were chosen to analyze the level of SA provided by G- 


BOSS, based on elements of the battlefield that are uncontrollable: 


a. Number of Role-players per Team 
This is the number of role-players per team. 
b. Role-player Stealth 
This is the level of stealth associated with each role-player. 
Cc. Role-player Speed 
This is maneuver speed for each role-player. 
C. NEARLY ORTHOGONAL LATIN HYPERCUBES (NOLH) 


NOLHs are a space-filling experimental design technique developed by 
COL Thomas Cioppa, United States Army, at the Naval Postgraduate School (NPS) in 
2002. This technique allows for the exploration of a large number of input parameters in 
an efficient number of runs, while maintaining nearly orthogonal design columns (Cioppa 
& Lucas, 2007). A design of 13 factors at just two levels, each using a full-factorial 
approach, would require nearly a quarter of a million runs (8,192 design points x 30 reps 
= 245,760 total runs). Using a crossed design with a stacked NOLH for the quantitative 
factors, the number of runs 1s reduced significantly to 23,760 (792 design points x 30 reps 
= 23,760) runs and the quantitative factors are more extensively varied. Figure 8 shows 
the NOLH design spreadsheet used for this analysis and Figure 9 shows the space-filling 
property of the NOLH technique. 
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E9 Microsoft Excel - Trial15NOLH_stack?.xls 

ad | File Edit Yiew Insert Format Tools Data Window Help 

one Oe ee ne eee See Weer mY ane A Bee 2 ooh Fi.) i 7 @ 
3 Aa ab| | [°""| as = Yi = i ae | 2 a r=) | aI 2 @ Security.. | Pras eZ | GO 


= level 112.5 7 01 142.5 0.01 12 0.01 0.01 
high level 187.5 0.99 237.5 0.99 180 0.99 a 0.99 : ; 
decimals 0 2 0 2 0 2 0 2 0 0 
factor name ClassRng n1PClass ClassRng n2PClass mLatency Reliability titySpeed ityStealth umPerTm ‘SlewRate 


N 


NIN NINN =\= 








Figure 8. NOLH Design Spreadsheet. 
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Figure 9. Scatterplot matrix of experiment factors. 
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Another advantage of NOLH is the negligible correlation among experiment 
factors, which prevents factor effects being confounded with one another. Figure 10 


shows minimal multicollinearity (negligible correlation amongst the experiment factors). 


Correlations 


Cam1ClassRng Cam1PClass Cam2ClassRng Cam2PClass CommLatency CommReliability EntitySpeed EntityStealth NumPerTm Tw rSlew Rate 
Cam1ClassRng 1.0000 -0.0020 -0.0032 0.0025 -0.0049 -0.0032 0.0020 -0.0031 -0.0044 
Cam1PClass -0.0020 1.0000 -0.0018 -0.0115 -0.0026 0.0039 -0.0031 0.0026 0.0219 
Cam2ClassRng -0.0032 -0.0018 1.0000 0.0015 -0.0061 -0.0137 -0.0113 0.0046 0.0033 
Cam2PClass 0.0025 -0.0115 0.0015 1.0000 0.0017 -0.0000 -0.0261 0.0074 0.0111 
CommLatency -0.0049 -0.0026 -0.0061 0.0017 1.0000 0.0039 -0.0050 -0.0039 0.0217 
CommReliability -0.0032 0.0039 -0.0137 -0.0000 0.0039 1.0000 0.0005 0.0115 -0.0149 
Entity Speed 0.0020 -0.0031 -0.0113 -0.0261 -0.0050 0.0005 1.0000 -0.0219 0.0252 
Entity Stealth -0.0031 0.0026 0.0046 0.0074 -0.0039 0.0115 -0.0219 1.0000 -0.0063 
NumPerTm -0.0044 0.0219 0.0033 0.0111 0.0217 -0.0149 0.0252 -0.0063 1.0000 
Tw rSlew Rate -0.0181 -0.0131 -0.0482 -0.0810 -0.0141 -0.0304 0.1074 0.1257 -0.0358 


650 rows not used due to missing or excluded values or frequency or w eight 
variables missing, negative or less than one. 





Figure 10. Experiment factor pairwise correlation matrix. 
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IV. DATA ANALYSIS 


After defining the problem, scenario, MOEs and experimental design, the 
simulation is executed to generate output data. The next task is the analysis of the data 
generated by data farming the simulation model. The analysis provided in this chapter 1s 
the result of an iterative and exhaustive process of applying various statistical techniques 
to the simulation output. Not every step of that exhaustive process is detailed in this 
section. Models and techniques resulting in significant findings about G-BOSS 
operational effectiveness are included in this chapter. JMP Statistical Discovery 
Software Version 7 is the primary tool used for this analysis. Details of this data analysis 
software are readily available at www.jmp.com. 

The analysis evaluates three specific research questions about enhancing the 


operational effectiveness of G-BOSS: 


e What critical factors determine the level of SA provided by G-BOSS” 
° Does coordination via the COC improve the effectiveness of G-BOSS? 


® Does use of motion detection radar improve the effectiveness of G-BOSS? 


Each question is addressed in this chapter, along with an explanation of the analysis 


technique and results. 


A. DATA SUMMARY 


A summary of the MOE data is provided in Figures 10 and 11. For clarity of 
presentation, the means of the 780 design points are graphed. The data summary includes 
the distribution data and 95% confidence intervals for the MOEs. Based on the 
distribution of the Mean-[ED-Detected MOE, the IED emplacement was detected 76% of 
the time. The Mean-Percent-Correct-Identifications MOE distribution data shows 
G-BOSS provides an average of 72% correct identifications in SA trial #15. Neither 
MOE data fits a Normal distribution. A Normal distribution of the data is not expected 


or required. 


pa 


Mean(PercCorrectID(3hits)) Moments 


Mean 72.91662 
Std Dev 21.810319 
Std Err Mean 0.7809347 


upper 95% Mean 74.449606 
lower 95% Mean 71.383634 
N 780 


Fitted Normal 
Parameter Estimates 


Type Parameter Estimate Lower 95% Upper 95% 
Location wu 72.91662 71.383634  74.449606 
Dispersion o 21.810319  20.779092 22.950053 


Goodness-of-Fit Test 


Shapiro-Wilk W Test 
-4 W Prob<W 
0.876874 0.0000* 





Normal Quantile Plot 


Note: Ho = The data is from the Normal distribution. Small p-values reject 
—— Normal(72.9166,21.8103) 


Ho. 
Figure 11. Data summary for the Mean-Percent-Correct-Identifications MOE. 
Mean(DetectlED) moma = 
001 .01 .0510 25 50 75 9095 .99 999 Mean 0.7669231 
i Std Dev 0.1397978 
Std Err Mean 0.0122611 


upper 95% Mean 0.7911819 
lower 95% Mean 0.7426642 
N 130 


Fitted Normal 
Parameter Estimates 


Type Parameter Estimate Lower 95% Upper 95% 
Location wu 0.7669231 0.7426642 0.7911819 
Dispersion o 0.1397978 0.1246219 0.1592157 


Goodness-of-Fit Test 
Shapiro-Wilk W Test 





WwW Prob<W 
Normal Quantile Plot 0.962815 0.0013* 
—— Normal(0.76692,0.1398) Note: Ho = The data is from the Normal distribution. Small p-values reject 
Ho. 
Figure 12. Data summary for the Mean-IED-Detected MOE. 


B. SA CRITICAL FACTORS 


Stepwise Regression and Classification and Regression Trees (CART) are the 
primary data analysis techniques used to identify the critical factors and interactions that 


determine the level of SA provided by G-BOSS in this scenario. 
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1. Stepwise Regression 


Regression is a common statistical technique used for investigating the effects of 
factors on a response variable (Montgomery, Peck, & Vining, 2006) or MOE. This 
technique is computationally intensive and allows the possibility to overfit a model. 
Stepwise regression reduces the parameter space to only those factors with a specified 
significance level by incrementally adding and deleting factors to the regression model. 
The result of a stepwise regression identifies the significant or critical factors that affect 
the response or MOE. Further, stepwise regression provides parameter estimates for the 
significant factors that result in the most preferred R-squared value. By definition, the 
R-squared value is the explained proportion of the variation in the response from fitting 
the model to the input values (Montgomery et al., 2006). 

The initial stepwise regression model for this study investigated all main effects 
and two-way interaction terms of the 13 experimental factors on the Mean-Percent- 
Correct-Identifications MOE as the response variable. A 0.05 level of significance is 
used in the development of this model. The results identified the most critical factors. 
Thus, the preferred regression model used in this study includes these seven critical 


factors and some of their interactions: 


e Distance Between Pairs of Towers 
® Number of Role-players per Team 
@ Role-player Speed 

e Tower Slew Rate 

e Number of COC Monitors 

® MSTAR/no MSTAR 

@ Role-player Stealth 


Figure 13 is a visual representation of the preferred model. The plot of the actual 
versus predicted response displays how closely the preferred model explains the MOE. 
This conclusion is made based on how well the data points follow the diagonal line. The 
residual by predicted plot shows that the assumption of heteroscedasticity, or an absence 


of a pattern in the residuals, is violated. Prediction is not the purpose of this model. The 
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purpose is gaining insight into the relative factor effects and their interactions. Thus, this 
finding of mild nonheteroscedasticity does not diminish any insights gained from 


this model. 


Actual by Predicted Plot Residual by Predicted Plot 


Residual 


Q 
= 
fo) 
O 
© 
= 
Zz 
= 


MN(PerCorrlD) 


30 40 50 60 70 80 90 100 110 
MN(PerCorrlD) Predicted 30 40 50 60 70 80 90 100 110 


P<.0001 RSq=0.96 RMSE=4.6663 MN(PerCorrID) Predicted 





Figure 13. Predicted versus actual Mean-Percent-Correct-Identifications and 
associated residual plot verifying the absence of pattern in the residuals. 


The preferred model in this study yields an R-squared of 0.96: 96% of the 
variability in the mean percent of correct identifications is explained by the critical 
factors identified in the model. The analysis of variance section in Figure 13 shows that 
the model is highly significant. This conclusion is based on the extremely low p-value 


for the F statistic. 


Response MN(PerCorrl!ID) 


Summary of Fit 


RSquare 0.956869 
RSquare Adj 0.954225 
Root Mean Square Error 4.666343 
Mean of Response 72.91662 
Observations (or Sum Woats ) 780 


Analysis of Variance 


Sum of 
Source DF Squares Mean Square F Ratio 
Model 45 354579.86 7879.55 361.8664 
Error 15982.67 21.77 Prob>F 
C. Total 370562.53 O0.0000* 





Figure 14. The R-squared value and significance of the stepwise regression model. 
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The effect of each critical factor on the MOE is shown in Figure 15. The absolute 
magnitude of the ft-Ratio indicates the relative influence a factor has on the outcome of 
the MOE. The //4.0/ value for Distance Between Pairs of Towers [4] 1s the largest 
value in the table and, thus, has the most statistical significance in explaining the MOE. 
The Distance Between Pairs of Towers factor 1s categorical and takes on the value of 
2km or 4 km. This factor has a positively correlated relationship with the MOE. When 
Distance Between Pairs of Towers is set at 4 km, the MOE value produced in the 
simulation increases. This is contrasted with the second highest magnitude value of 
—22.88 for the Number of Role-players per Team. ‘This factor is a nonzero integer and is 
negatively correlated with the MOE (1.e., as Number of Role-players per Team increases, 
the MOE value decreases). Inspection of the remaining f-Ratio values yield an 
understanding of how the factors influence the MOE. The proportion of variance 
explained by each factor is shown in Figure 16. Again, the positioning of the towers is 


by far the most important factor. 
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Scaled Estimates 


Nominal factors expanded to all levels 


Continuous factors centered by mean, scaled by range/2 


Term 

Intercept 

COCs WithMon{O&1-2} 

COCs WithMon{O- 1} 

DistBetw Tw rs[2] 

DistBetw Tw rs[4] 

MSTAR[No] 

MSTARTY es] 

Cam1ClassRng 

Cam PClass 

Cam2ClassRng 

Cam2PClass 

CommLatency 

CommReliability 

Entity Speed 

Entity Stealth 

NumPerTm 

Tw rSlew Rate 

(COCs WithMon{O&1-2}-0.33333)*DistBetw Tw rs[2] 
(COCs WithMon{O&1-2}-0.33333)*DistBetw Tw rs[4] 
COCs WithMon{O- 1}*DistBetw Tw rs[2] 

COCs WithMon{O-1}*DistBetw Tw rs[4] 

(COCs WithMon{O&1-2}-0.33333)*MSTAR[No] 
(COCs WithMon{O&1-2}-0.33333)*MSTART[Y es] 
COCs WithMon{O- 1}*MSTAR[No] 

COCs WithMon{O-1}*MSTARTY es] 
COCsWithMon{O- 1}*(Entity Speed-8.50769) 


(COCs WithMon{O0&1-2}-0.33333)*(NumPertTn-4.06154) 


COCs WithMon{O- 1}*(NumPerTnm-4.06154) 
DistBetw Tw rs[2]*MSTAR[No] 

DistBetw Tw rs[2]*MSTARTY es] 

DistBetw Tw rs[4]*MSTAR[No] 

DistBetw Tw rs[4]*MSTARTY es] 

DistBetw Tw rs[2]*(Cam1 Class Rng- 150.031) 
DistBetw Tw rs[4]*(Cam1 Class Rng- 150.031) 
DistBetw Tw rs[2]*(Cam1PClass-5003.08) 
DistBetw Tw rs[4]*(Cam1PClass-5003.08) 
DistBetw Tw rs[2]*(Cam2PClass-5003.08) 
DistBetw Tw rs[4]*(Cam2PClass-5003.08) 
DistBetw Tw rs[2]*(CommLatency-64.0821) 
DistBetw Tw rs[4]*(CommLatency-64.0821) 
DistBetw Tw rs[2]*(Entity Speed-8.50769) 

DistBetw Tw rs[4]*(Entity Speed-8.50769) 

DistBetw Tw rs[2]*(Entity Stealth-O.50031) 
DistBetw Tw rs[4]*(Entity Stealth-O.50031) 
DistBetw Tw rs[2]*(NumPerTm4.061 54) 

DistBetw Tw rs[4]*(NumPerTn+4.061 54) 

DistBetw Tw rs[2]*(Tw rSlew Rate-2.03077) 
DistBetw Tw rs[4]*(Tw rSlew Rate-2.03077) 
MSTAR[No]*(CommReliability-50.0308) 

MSTARTY es]*(CommReliability-50.0308) 
MSTAR[No]*(Entity Speed-8.50769) 

MSTARTY es]*(Entity Speed-8.50769) 
MSTARI[No]*(NumPerTn4.06154) 

MSTARTY es]*(NumPerTm-4.06154) 
MSTAR[No]*(Tw rSlew Rate-2.03077) 

MSTARTY es]*(Tw rSlew Rate-2.03077) 

(Cam1 ClassRng- 150.031 )*(Cam2ClassRng-190.031) 
(Cam1 ClassRng- 150.031 )*(Entity Speed-8.50769) 
(Cam1 ClassRng- 150.031 )*(Entity Stealth-O.50031) 
(Cam1 PClass-5003.08)*(CommReliability-50.0308) 
(Cam1 PClass-5003.08)*(Tw rSlew Rate-2.03077) 
(Cam2ClassRng- 190.031 )*(Cam2PClass-5003.08) 
(Cam2PClass-5003.08)*(CommLatency-64.0821 ) 
(CommLatency-64.0821 )*(Entity Speed-8.50769) 

(Entity Speed-8.50769)*(Entity Stealth-O.50031) 


Figure 15. 


Scaled 
Estimate 
73.164831 
-2.527669 
-1.668293 
-19.04862 
19.048616 
-2.272248 
2.2722483 
1.3094355 
1.8591256 
0.6913811 
2.1100025 
-1.083035 
-0.044681 
5.2992629 
-3.037041 
-6.076454 
3.8061006 
-2.346151 
2.3461505 
-0.965988 
0.9659878 
2.5304753 
-2.530475 
1.5296904 
-1.52969 
0.492062 
1.7237415 
1.6956608 
-2.151542 
2.1515423 
2.1515423 
-2.151542 
-0.618336 
0.6183355 
-1.496109 
1.4961085 
-1.514234 
1.514234 
1.2360203 
-1.23602 
1.1528621 
-1.152862 
1.5891095 
-1.58911 
-2.174915 
2.174915 
1.5164155 
-1.516416 
-0.567596 
0.5675963 
1.3529832 
-1.352983 
3.457763 
-3.457763 
0.5327884 
-0.532788 
-2.359057 
-1.3098 
0.8555647 
-1.570068 
1.5608138 
4.0188281 
1.5086946 
1.7999482 
3.3704508 


Std Error 
0.173026 
-264089 
-204633 
-167082 
.167082 
.167082 
.167082 
-280208 
-278335 
-276819 
.279189 
.366653 
-278941 
.275464 
-281187 
.265612 
0.23718 
-263831 
-263831 
-204633 
-204633 
177217 
.177217 
-204633 
-204633 
.334519 
-280892 
.324039 
167082 
.167082 
.167082 
.167082 
-279558 
-279558 
-278089 
-278089 
.279004 
-279004 
O.366 
0.366 
.274984 
.274984 
-280421 
-280421 
-265079 
-265079 
-235583 
-235583 
-278136 
-278136 
274757 
274757 
0.26481 
0.26481 
0.232862 
0.232862 
0.655455 
0.574471 
0.509648 
0.631957 
0.413492 
0.573359 
0.440134 
0.432194 
0.560603 


o0000000O000O0O0O0O0O0O0O0O00 0 oo0o0o0o0o0qo0o0o00 0000 


oo0o0o0o0o0o00 000 0 


t Ratio 
422.86 
-9.57 
-8.15 
-114.01 
114.01 
-13.60 
13.60 
.67 
.68 
.50 
.56 
.95 
.16 
.24 
.80 
.88 
.05 
.89 
.89 
72 
72 
.28 
.28 
.48 
.48 
47 
.14 
.23 
.88 
.88 
.88 
.88 
21 
21 
.38 
.38 
.43 
.43 
.38 
.38 
.19 
.19 
.67 
.67 
.20 
.20 
.44 
.44 
.04 
.04 
.92 
.92 
.O6 
.06 
.29 
.29 
.60 
.28 
.68 
.48 
77 
.O1 


Prob=|t| 


Oo 
< 
< 
Oo 
Oo 
< 
< 
< 
< 
Oo 
< 
Oo 
Oo 
< 
< 
< 
< 
< 
< 
< 
< 
< 
< 
< 
< 
Oo 
< 
< 
< 
< 
< 
< 


< 
Oo 
< 
< 


The f-ratio column explains the relationship between the factor and the MOE. 
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OODDODOOOAANNDOAANAANANAANKROOANAAAAOO 


-OO0O* 
-0001* 
-0001* 
-OOO0O* 
-OO0O0* 
-0001* 
-0001* 
-0001* 
-0001* 
.0127* 
-0001* 
-0032* 
.8728 

-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-1417 

-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-0273* 
-0273* 
-0001* 
-0001* 
-0001* 
-0001* 
-0008* 
-0008* 
-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-0001* 
-0416* 
-0416* 
-0001* 
-0001* 
-0001* 
-0001* 
-0224* 
-0224* 
OO0O3* 
-0229* 
-0936 

-0132* 
-0002* 
-0001* 
-O0OOO6* 
-0001* 
-0001* 





Scaled estimates of the preferred stepwise regression used in the study. 


Effect Tests 


Source 

COCs WithMon{O&1-2} 
COCs WithMon{O- 1} 
DistBetw Tw rs 
MSTAR 

Cam 1ClassRng 
Cam1PClass 
Cam2ClassRng 
Cam2PClass 
CommLatency 
CommReliability 
Entity Speed 

Entity Stealth 
NumPerTm 

Tw rSlew Rate 


COCs WithMon{O&1-2}*DistBetw Tw rs 


COCs WithMon{O-1}*DistBetw Tw rs 
COCs WithMon{O&1-2}*MSTAR 
COCs WithMon{0-1}*MSTAR 
COCs WithMon{O-1}*Entity Speed 
COCs WithMon{O&1-2}*NumPer Tm 
COCs WithMon{O-1}*NumPerTm 
DistBetw Tw rs*MSTAR 
DistBetw Tw rs*Cam1ClassRng 
DistBetw Tw rs*Cam1PClass 
DistBetw Tw rs*Cam2PClass 
DistBetw Tw rs*CommLatency 
DistBetw Tw rs*EntitySpeed 
DistBetw Tw rs*Entity Stealth 
DistBetw Tw rs*NumPerTm 
DistBetw Tw rs*Tw rSlew Rate 
MSTA R*CommReliability 

MSTA R*Entity Speed 
MSTAR*NumPer Tm 

MSTAR*Tw rSlew Rate 

Cam1 Class Rng*Cam2ClassRng 
Cam1 Class Rng*Entity Speed 
Cam1 Class Rng*Entity Stealth 
Cam 1 PClass*CommReliability 
Cam1PClass*Tw rSlew Rate 
Cam2ClassRng*Cam2PClass 
Cam2PClass*CommLatency 
CommLatency“*Entity Speed 


Figure 16. 


=—a =—_ = = = = = = = = = 2 Sk Sk Sl Sl kl Sl ll ll Sl kl ll lll ll ll Sl lll ll lll lll lll] Ck 


|e) 
A 


=— =— = = = = = = = = 2 SS Sk Sl Sl ll ll ll lll kl lll ll lll ll ll lll ll ll ll ll lll lll] Ck 


Sum of 
Squares 
1994.78 
1447.26 
283022.82 
4027.23 
475.51 
971.48 
135.83 
1243.72 
189.99 
0.56 
8058.53 
2540.17 
11396.17 
5607.36 
1721.92 
485.23 
4439.63 
1216.78 
47.11 
820.01 
596.26 
3610.72 
106.53 
630.25 
641.38 
248.34 
382.73 
699.26 
1465.84 
902.20 
90.68 
528.01 
3712.57 
113.99 
282.06 
113.20 
61.36 
134.41 
310.26 
1069.79 
255.85 
377.67 


F Ratio 
91.6096 
66.4653 

12997.75 
184.9494 
21.8377 
44.6151 

6.2380 
57.1173 

8.7252 

0.0257 

370.0861 
116.6568 
523.3660 
257.5164 
79.0790 
22.2840 
203.8886 
55.8801 

2.1637 
37.6587 
27.3831 

165.8216 

4.8922 
28.9440 
29.4554 
11.4048 
17.5768 
32.1135 
67.3183 
41.4331 

4.1645 
24.2487 

170.4988 
5.2349 
12.9536 
5.1985 

2.8182 

6.1725 
14.2485 
49.1299 
11.7499 
17.3446 


Prob >F 
<.0001* 
<.0001* 
O0.0000* 
<.0001* 
<.0001* 
<.0001* 
0.0127* 
<.0001* 
0.0032* 
0.8728 
<.0001* 
<.0001* 
<.0001* 
<.0001* 
<.0001* 
<.0001* 
<.0001* 
<.0001* 
0.1417 
<.0001* 
<.0001* 
<.0001* 
0.0273* 
<.0001* 
<.0001* 
O0.0008* 
<.0001* 
<.0001* 
<.0001* 
<.0001* 
0.0416* 
<.0001* 
<.0001* 
0.0224* 
O0.0003* 
0.0229* 
0.0936 
0.0132* 
O0.0002* 
<.0001* 
O0.0006* 
<.0001* 





Sum of squares of the preferred stepwise regression used in the study. The 


sum of squares column explains the proportion of variance explained by each factor in 
the stepwise regression model. 
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Figure 17 shows an interaction plot of factors included in the preferred regression 
model. An interaction indicates that the change in the MOE caused by varying one 
parameter is dependent upon the value of another parameter. In an interaction plot, the 
MOE is on the y-axis and the factors involved in the interaction are on the x-axis or 
appear as separate lines as indicated in the legend. The trellis plot containing the 
interactions shows the high and low levels of the factor on the row and the trend in the 
MOE by changing the factor in the column. For example, Figure 17 (box all the way to 
the right in the second row) shows the interaction between Role-player Stealth and Tower 
Slew Rate. This graphic indicates: When Role-player Stealth is very high, the Tower 
Slew Rate has only marginal effect on the MOE; however, when Stealth is low, 


increasing the Tower Slew Rate improves the Mean-Percent-Correctly-Identified. 
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Figure 17. Interaction plots of four critical factors that determine the level of SA. 
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Figure 18 is a contour plot that reinforces the effect of the interaction between 
Tower Slew Rate and Role-player Stealth has on the MOE. In this plot, the slower 
G-BOSS slews its field-of-regard, the less stealth is required to avoid detection. The 
stealth level used by snipers and IED emplacers affects the MOE regardless of the 
G-BOSS slew rate. 


Contour Plot for MN(PerCorrlID) 


MIN(PerCorrID) 

mM <= 60.000 
mm <= 62.500 
“<= 65.000 
MS <= 67.500 
MS <= 70.000 
MS <= 72.500 
MS <= 75.000 
MS <= 77.500 
MS > 77.500 


0.0 0.1 0.2 0.3 0.4 0.5 0.7 0.8 0.9 1.0 


Entity Stealth 





Figure 18. Contour plot of interaction between Tower Slew Rate and Role-player 
Stealth, and their effect on the MOE. 


Zi Regression Trees 


Regression trees are another effective tool used to analyze the relationship 
between factors and MOEs. A regression tree is a recursive partition of the raw data into 
sets of inputs containing similar responses. Partitioning of the data occurs successively, 
according to the optimal splitting value determined from all possible values of each 
available variable. Figure 19 displays a recursive split of the data from 780 MANA runs 


on all of the experimental factors for the Mean-Percent-Correct-Identifications MOE. 
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Partition for MN(PerCorrlID) 


Number 





RSquare N of Splits 
0.808 780 5 













780 LogWorth Difference 
Mean 72.91662 546.03322 38.0972 
Std Dev 21.810319 


DistBetw Twrs( ] DistBetw Twrs(4) 


Count 390 390 LogWorth Difference 









Mean 53.868004 Mea 91.965236 60.053335 16.052 
Std Dev 12.814682 - Std Dev 7.7988301 







EntitySte alth<0.93 
Count 354 LogWorth Difference 
Mean 93.446956 31.209192 6.8505 
Std Dev 6.3604373 


ath >=0.93 
36 
Wiean 77.394991 


Std Dev 5.2352386 


Cam 2PClass<2900 Cam 2PClass>=2900 
Count 108 LogWorth Difference Count 246 LogWorth Difference 
Mean 88.686438  22.85835 18.1722 Mean 95.536939  26.305429 3.82769 
Std Dev 8.5728871 Std Dev 3.437301 


Cam1ClassRng<127 Cam1ClassRng>=127 EntitySpeed<7 EntitySpeed>=7 
Count 12 Count 96 Count 156 
Mean 72.533333 Mean 90.705576 Mean 93.109626 || Mean 96.937312 
Std Dev 8.1279751 Std Dev 6.1684583 Std Dev 3.9466573 || Std Dev 2.082526 





Figure 19. Regression tree split on the raw data by Mean-Percent-Correct- 
Identifications of Correct Identifications. 


This analysis complements the regression analysis in the previous section. The 
first split is made on Distance Between Pairs of Towers. ‘This is the single most 
significant factor identified. The regression tree provides the optimal split point at 2 km 
and 4 km. G-BOSS towers placed 4 km apart provide a mean percent of correctly 


identified role-players nearly two times greater than G-BOSS towers placed 2 km apart. 


The next most important factor is Role-player Stealth. When G-BOSS towers are 
placed 4 km apart and a sniper or IED emplacer’s stealth is less than 0.93, the expected 
percent of correct identifications 1s 93%. If Role-player Stealth is greater than or equal to 
0.93, then the percent of correct identifications drops to 77%. Although the particular 


value of 0.93 does not translate directly to the real world, one of the key “uncontrollable 
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factors” 1s highly significant. This is a notable result. Perhaps countermeasures to enemy 
stealth can offset this effect. For example, training G-BOSS operators to a higher level of 
vigilance and detailed classification of “suspicious” behavior may be _ suitable 


countermeasures. Other significant factors identified by the regression tree include: 


e Camera 2 P(Classification) 
O (best when greater than .29) 
e Camera I Classification Range 
O (best when greater than 127 model grid squares, which is roughly 


equivalent to 2.5 km) 
® Role-player Speed 


O (best when they are moving at speeds greater than 7/100 grid 
Squares per time step, which is roughly equivalent to | km/hr) 


C; COC COORDINATION VERSUS STAND-ALONE COMPARISONS 


This section examines whether the simulation results of a coordinated 
configuration of G-BOSS improved operational effectiveness. This analysis is based on 
both MOEs. The following comparisons are conducted: 

® Coordinated configuration of G-BOSS against stand-alone configuration 

without MSTAR (distance between towers 1s set at 2 km). 


e Coordinated configuration of G-BOSS against stand-alone configuration 
with MSTAR (distance between towers is set at 2 km). 


The MOE data does not fit a Normal distribution. Thus, a nonparametric statistical 
technique is used. The Wilcoxon Signed Ranks Test is designed to test whether a 
particular sample came from a population with a specified mean or median 


(Conover, 1999). 


1. Coordinated versus Stand-Alone Configurations without MSTAR 


This analysis is conducted as a hypothesis test. The null hypothesis is that the 
differences between two independent populations of simulation data from a coordinated 
G-BOSS configuration and a stand-alone configuration have the same mean. The 


alternative hypothesis is that the means are not equal. The notation used for this test is: 


oy, 


Mean(PercentCorrectlD ) coordinated _ Configuration without _ MSTAR ae 


Mean(PercentCorrecilD ) 5 isn Atone_ Configuration without _MSTAR - G 


D=F-G 
H,:E(D)=0 
H,:E(D)#0 


Figures 20 and 21 show that without the presence of MSTAR, the means between the 
coordinated G-BOSS approach and the stand-alone G-BOSS configuration are not equal. 
There is evidence to reject the null hypothesis at a 0.05 or 0.10 level of significance. This 
conclusion is based on the p-value (0.0001). There is a statistically significant increase in 
performance between the networked configuration and the stand-alone configuration. 
This finding is due to an increase of approximately two percentage points between the 
two configurations. The small difference between the two configurations is so small that 
it should not be considered a practical significant difference. In other words, there is not 
enough evidence to deem the coordinated configuration more effective than the 


stand-alone configuration, based on the marginal increase in performance. 
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70 


60 


Mean(PercCorrectID(3hits )) 
On 
Oo 


COCs WithMon 





2Mon-Mean(PercCorrectID) 50.6272 t-Ratio 5.17721 


OMon-MeanPercCorrectID) 48.632 DF 64 
Mean Difference 1.99504 b>|t} §<.0001* 
Std Error 0.38535 Prob > <.0001* 
Upper95% 2.16487 Prob <t 0000 


Low er95% 1.22521 
Wilcoxon Sign-Rank 


2Mon-Mean(PercCorrectID)- 
OMon-MeanPercCorrectID) 





Test Statistic 705.500 
Prob > |z| <.0001* 

Prob > z <.0001* 

Prob < z 1.0000 


Figure 20. Wilcoxon SignedRank Test results for coordinated G-BOSS configuration 
against stand-alone G-BOSS configuration for the Mean-Percent-Correct-Identification 
MOE. (No MSTAR) 
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Test ChiSquare Prob>ChiSq 
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Pearson 1.714 0.1904 


Figure 21. Wilcoxon Signed Rank Test results for coordinated G-BOSS configuration 
against stand-alone G-BOSS configuration for the Mean-I[ED-Detected MOE. 
(No MSTAR) 


2. Coordinated versus Stand-alone Configurations with MSTAR 
This analysis is conducted as a hypothesis test in a similar manner: 


Mean(PercentCorrectID ) Coordinated _ Configuration with _MSTAR a 
Mean(PercentCorrectID ) stand _ Alone _ Configuration with _MSTAR ~G 
D=F-G 

H,:E(D)=0 

H,:E(D)#0 


Figures 22 and 23 show that with the presence of MSTAR, the means between the 
coordinated G-BOSS approach and the stand-alone G-BOSS configuration are not equal. 
Thus, there is enough evidence to reject the null hypothesis at a 0.05 or 0.10 level of 
significance. This conclusion is based on the low p-value (0.0001). There is both a 
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statistically and practically significant increase in performance with the networked 
configuration as compared to the stand-alone configuration. The performance with the 
networked configuration increases by nearly 25% (from approximately 47% correct 
classifications to about 70% correct classifications). This increase is significant enough 
to deem the coordinated configuration more effective than the stand-alone configuration 


based on the marginal increase in performance. 
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COCsWithMon 
2Mon-Mean(PercCorrectID) 69.8949 t-Ratio 12.73921 
OMon-Mean(PercCorrectID) 47.4422 DF 64 
Mean Difference 22.4527 >|t| = <.0001* 
Std Error 1.76249 Prob > <.0001* 
Upper95% 25.9737 Prob <t .0000 






Low er95% 18.9318 
N 65 
Correlation 0.4191 


Wilcoxon Sign-Rank 


2Mon-Mean(PercCorrectID)- 
OMon-Mean(PercCorrectID) 





Test Statistic 1069.50 
Prob > |z| <.0001* 

Prob >z <.0001* 

Prob <z 1.0000 


Figure 22. Wilcoxon Signed Rank Test results for coordinated G-BOSS configuration 
against stand-alone G-BOSS configuration (with MSTAR). 
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| Contingency Analysis of DetectlIED By #Monitors 
Mosaic Plot | 


fal 
a 
is) 
= 
oO 
fal 





#Monitors 


N DF -LogLike RSquare (U) 
60 1 7.9382475 0.2936 
Test ChiSquare Prob>ChiSq 
Likelinood Ratio 15.876 <.0001* 
Pearson 12.000 0.0005* 
Figure 23. 


Wilcoxon Signed Rank Test results for coordinated G-BOSS configuration 
against stand-alone G-BOSS configuration for the Mean-IED-Detected MOE. 
(With MSTAR) 
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V. CONCLUSIONS 


This research set out to determine how Marines deployed now can best employ 


G-BOSS. An operational test of G-BOSS conducted by MCOTEA is the basis for the 


simulation model created in this study. This thesis produced a comparative, quantitative 


analysis of G-BOSS employment TTPs and the critical factors that determine the level of 


SA provided by G-BOSS. The results of this thesis provide insights to enhance 


operational effectiveness of G-BOSS. Furthermore, the simulation built for this research 


can serve as the foundation for many additional studies. 


A. ANALYSIS INSIGHTS 


The insights gained in this analysis are: 


The positioning of the towers is the most critical factor associated with 
enhancing the operational effectiveness of G-BOSS. A distance of 4 km 
between G-BOSS towers results in a proportion of correct identifications 
of 0.91 while a distance of 2 km results in a proportion of 0.53. When 
G-BOSS is employed in open terrain, more dispersion results in 
better performance. 


The stealth of the role-players has a significant effect on the proportion of 
correct identifications. This result is intuitive and helps validate the 
model, since snipers and IED emplacers tend to use stealth to mask their 
hostile acts or intent. Increased training of G-BOSS operators’ level of 
vigilance is a countermeasure to mitigate enemy stealth. 


The interaction between tower slew rate and stealth of the role-players 1s 
significant. A faster G-BOSS slew rate provides a greater proportion of 
correct identifications when role-players’ stealth is low. | When 
role-players’ stealth is high, a low proportion of correct identifications is 
achieved, regardless of tower slew rate. Recall that tower slew rate is the 
model’s surrogate for the speed at which G-BOSS operators can observe a 
field-of-regard. 


Without the presence of MSTAR, the coordinated G-BOSS configuration 
produces a slightly larger proportion of correct identifications than the 
stand-alone G-BOSS _ configuration. The coordinated G-BOSS 
configuration results in an overall proportion of correct identification of 
0.51, while the stand-alone G-BOSS configuration results in an overall 
proportion of correct identification of 0.49. The results of the comparison 
between the G-BOSS configurations are not practically significant. This 
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is based on the marginal increase of approximately two percentage points 
between the coordinated G-BOSS configuration and the stand-alone G- 
BOSS configuration. Further, this finding answers the question of 
underutilization: Stand-alone employment of G-BOSS 1s nearly as 
effective as a coordinated employment of G-BOSS without MSTAR. 


With MSTAR, the coordinated G-BOSS configuration produces a 
significantly larger proportion of correct identifications than what the 
stand-alone configuration produces. The coordinated G-BOSS 
configuration results in an overall proportion of correct identifications of 
0.71 and the stand-alone configuration results in an overall proportion of 
correct identifications of 0.47. This result is practically and statistically 
significant. MSTAR facilitates the detection of more agents. MSTAR 
coupled with data fusion at the COC affords commanders an 
enhanced capability. 


The emplacement of the IED was detected in 76% of all of the simulation 
excursions conducted (which varied 13 factors associated with GBOSS or 
the role-players). This is promising since G-BOSS’s mission is to counter 
the threat of IEDs. 


FOLLOW-ON WORK 


The following is a list of follow on research of value stemming from this research: 


Additional validation and comparison of the MANA simulation results 
with the results of MCOTEA’s live experiment. 


Further research into the utility of using simulation to develop synthetic 
operational experiences in new doctrine and tactics. 


Human factors study on how the vigilance of G-BOSS operators affects 
the level of SA provided by G-BOSS. 


Application of the findings in this thesis on enhancing the operational 
effectiveness of G-BOSS in an urban environment. 


Research into the operational effectiveness provided by G-BOSS when 
integrated with various organic assets (UAS, C2 assets, and Fire 
Support Systems). 
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